Introduction to Computer Vision: Plant Seedlings Classification¶

Problem Statement¶

Context¶

In recent times, the field of agriculture has been in urgent need of modernizing, since the amount of manual work people need to put in to check if plants are growing correctly is still highly extensive. Despite several advances in agricultural technology, people working in the agricultural industry still need to have the ability to sort and recognize different plants and weeds, which takes a lot of time and effort in the long term. The potential is ripe for this trillion-dollar industry to be greatly impacted by technological innovations that cut down on the requirement for manual labor, and this is where Artificial Intelligence can actually benefit the workers in this field, as the time and energy required to identify plant seedlings will be greatly shortened by the use of AI and Deep Learning. The ability to do so far more efficiently and even more effectively than experienced manual labor, could lead to better crop yields, the freeing up of human inolvement for higher-order agricultural decision making, and in the long term will result in more sustainable environmental practices in agriculture as well.

Objective¶

The aim of this project is to Build a Convolutional Neural Netowrk to classify plant seedlings into their respective categories.

Data Dictionary¶

The Aarhus University Signal Processing group, in collaboration with the University of Southern Denmark, has recently released a dataset containing images of unique plants belonging to 12 different species.

  • The dataset can be download from Olympus.
  • The data file names are:
    • images.npy
    • Label.csv

  • Due to the large volume of data, the images were converted to the images.npy file and the labels are also put into Labels.csv, so that you can work on the data/project seamlessly without having to worry about the high data volume.

  • The goal of the project is to create a classifier capable of determining a plant's species from an image.

List of Species

  • Black-grass
  • Charlock
  • Cleavers
  • Common Chickweed
  • Common Wheat
  • Fat Hen
  • Loose Silky-bent
  • Maize
  • Scentless Mayweed
  • Shepherds Purse
  • Small-flowered Cranesbill
  • Sugar beet

Note: Please use GPU runtime to execute the code efficiently¶

Importing necessary libraries¶

In [71]:
#Reading the training images from the path and labelling them into the given categories
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import cv2 # this is an important module to get imported which may even cause issues while reading the data if not used
import seaborn as sns # for data visualization
import tensorflow as tf
import keras
import os
# Library to split data
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder

from tensorflow.keras.models import Sequential #sequential api for sequential model
from tensorflow.keras.layers import Dense, Dropout, Flatten #importing different layers
from tensorflow.keras.layers import Conv2D, MaxPooling2D, BatchNormalization, Activation, Input, LeakyReLU,Activation
from tensorflow.keras import backend
from tensorflow.keras.utils import to_categorical #to perform one-hot encoding
from tensorflow.keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D
from tensorflow.keras.optimizers import RMSprop,Adam,SGD #optimiers for optimizing the model
from tensorflow.keras.callbacks import EarlyStopping  #regularization method to prevent the overfitting
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras import losses, optimizers

# Importing all the required sub-modules from Keras
from keras.applications.vgg16 import VGG16
from keras.preprocessing.image import ImageDataGenerator
from keras.preprocessing.image import img_to_array, load_img

Loading the dataset¶

In [72]:
# Mount Google drive to access the dataset
# Run the below code if you using google colab
from google.colab import drive
drive.mount('/content/drive')

Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
In [73]:
# Reading the dataset
images = np.load("/content/drive/MyDrive/images.npy")
labels = pd.read_csv("/content/drive/MyDrive/Labels.csv")

Data Overview¶

Understand the shape of the dataset¶

In [74]:
# Printing the images shape
images.shape
Out[74]:
(4750, 128, 128, 3)
  • There are 4750 images of size 128 * 128
In [75]:
# Printing the labels shape
labels.shape
Out[75]:
(4750, 1)
  • Labels has 4750 entries in one column

Exploratory Data Analysis¶

  • EDA is an important part of any project involving data.
  • It is important to investigate and understand the data better before building a model with it.
  • A few questions have been mentioned below which will help you understand the data better.
  • A thorough analysis of the data, in addition to the questions mentioned below, should be done.
  1. How are these different category plant images different from each other?
  2. Is the dataset provided an imbalance? (Check with using bar plots)
In [76]:
# Count in each category
label_counts = labels['Label'].value_counts(normalize=True)
print (label_counts)

Loose Silky-bent             0.137684
Common Chickweed             0.128632
Scentless Mayweed            0.108632
Small-flowered Cranesbill    0.104421
Fat Hen                      0.100000
Charlock                     0.082105
Sugar beet                   0.081053
Cleavers                     0.060421
Black-grass                  0.055368
Shepherds Purse              0.048632
Common wheat                 0.046526
Maize                        0.046526
Name: Label, dtype: float64
In [77]:
# Plotting bar chart to see the data
label_counts.plot(kind='bar')
plt.title('Label Counts')
plt.xlabel('Label')
plt.ylabel('Count')
plt.show()
  • Maize, Common wheat and Shepeherds Purse have less data
  • Loose Silky-bent, Common Chickweed and Scentless Mayweed has the most data

Let's visualize images randomly from each of the twelve classes.

In [78]:
# Generate random indices for each class
unique_labels = np.unique(labels)
num_classes = len(unique_labels)
In [79]:
# Generate random indices for each class
random_indices = []

for i in range(num_classes):
    indices_for_class = np.where(labels == unique_labels[i])[0]

    # Check if there are samples for the current class
    if len(indices_for_class) > 0:
        random_index = np.random.choice(indices_for_class)
        random_indices.append(random_index)
    else:
        print(f"Class {i} has no samples.")

# Print the random indices for each class
for i, index in enumerate(random_indices):
    print(f"Class {i}: Random Index {index}")

Class 0: Random Index 4059
Class 1: Random Index 2203
Class 2: Random Index 2457
Class 3: Random Index 1836
Class 4: Random Index 1214
Class 5: Random Index 636
Class 6: Random Index 4369
Class 7: Random Index 3771
Class 8: Random Index 2749
Class 9: Random Index 1016
Class 10: Random Index 28
Class 11: Random Index 3544
In [80]:
# Plot random images from each class
fig, axes = plt.subplots(4, 3, figsize=(15, 15))
axes = axes.flatten()
for i, index in enumerate(random_indices):
    image = images[index]
    label= labels.loc[index]
    # Assuming images are in RGB format
    axes[i].imshow(image)
    axes[i].set_title(f'Class {label}')
    axes[i].axis('off')

plt.show()
In [81]:
# Printing one individual  image
plt.imshow(images[1010])
Out[81]:
<matplotlib.image.AxesImage at 0x780ba149bd00>

Convert the BGR images to RGB images.¶

In [82]:
# Convert the images to RGB from BGR

rgb_images = images[..., ::-1]
In [83]:
# Check the shape of the RGB images
rgb_images.shape
Out[83]:
(4750, 128, 128, 3)
In [84]:
# Plot random RGB images from each class
fig, axes = plt.subplots(4, 3, figsize=(15, 15))
axes = axes.flatten()
for i, index in enumerate(random_indices):
    image = rgb_images[index]
    label= labels.loc[index]
    # Assuming images are in RGB format
    axes[i].imshow(image)
    axes[i].set_title(f'Class {label}')
    axes[i].axis('off')

plt.show()
In [85]:
# Plotting one RGB image
plt.imshow(rgb_images[1010])
Out[85]:
<matplotlib.image.AxesImage at 0x780ba14995d0>

Resize the images¶

As the size of the images is large, it may be computationally expensive to train on these larger images; therefore, it is preferable to reduce the image size from 128 to 64.

In [86]:
# First I trid to resize the images to 64 x 64 but did not get good result.
# so switched and resize the rgb images back to 128 x 128
resized_rgb_images = []

# Iterate through each image and resize
for image in rgb_images:
    # Resize the image using OpenCV
    resized_rgb_image = cv2.resize(image, (128, 128))

    # Append the resized image to the list
    resized_rgb_images.append(resized_rgb_image)

# Convert the list of resized images to a NumPy array
resized_rgb_images = np.array(resized_rgb_images)
In [87]:
# Check the images shape for resize
resized_rgb_images.shape
Out[87]:
(4750, 128, 128, 3)

Data Preparation for Modeling¶

  • Before you proceed to build a model, you need to split the data into train, test, and validation to be able to evaluate the model that you build on the train data
  • You'll have to encode categorical features and scale the pixel values.
  • You will build a model using the train data and then check its performance

Split the dataset

In [88]:
# Let us train and test the
resized_rgb_images_2d = resized_rgb_images.reshape(resized_rgb_images.shape[0], 128,128,3)
print(resized_rgb_images_2d.shape)
print(labels.shape)
X_train, X_test, y_train, y_test = train_test_split(resized_rgb_images_2d
,labels , test_size=0.1, random_state=1,stratify=labels)

(4750, 128, 128, 3)
(4750, 1)
In [89]:
# Converting the list into DataFrame
y_train = pd.DataFrame(y_train, columns=["Label"],dtype=object)
y_test = pd.DataFrame(y_test, columns=["Label"],dtype=object)

Data Normalization¶

In [90]:
# Normalizing the image pixels
X_train = X_train.astype('float32')/255.0
X_test = X_test.astype('float32')/255.0

Encoding Target Variable¶

In [91]:
# Let us encode using Label Encoder
label_encoder = LabelEncoder()

# Fit the label encoder on the combined set of labels
all_labels = np.concatenate([y_train, y_test])
label_encoder.fit(all_labels)


# Transform labels to integers
y_train_e = label_encoder.transform(y_train)
y_test_e = label_encoder.transform(y_test)

/usr/local/lib/python3.10/dist-packages/sklearn/preprocessing/_label.py:99: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)
/usr/local/lib/python3.10/dist-packages/sklearn/preprocessing/_label.py:134: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, dtype=self.classes_.dtype, warn=True)
/usr/local/lib/python3.10/dist-packages/sklearn/preprocessing/_label.py:134: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, dtype=self.classes_.dtype, warn=True)
In [92]:
# Creating one-hot encoded representation of target labels
# We can do this by using this utility function - https://www.tensorflow.org/api_docs/python/tf/keras/utils/to_categorical
# to_categorical() function is also explained in the Neural Networks Module

y_train_encoded_oh = tf.keras.utils.to_categorical(y_train_e)
y_test_encoded_oh = tf.keras.utils.to_categorical(y_test_e)
y_train_e = tf.keras.utils.to_categorical(y_train_e, num_classes=12)
y_test_e = tf.keras.utils.to_categorical(y_test_e,num_classes=12)

Model Building¶

Model 1 Limited Layers

In [93]:
from tensorflow.keras import backend
backend.clear_session()
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
In [94]:
# Initializing a sequential model
model = Sequential()
# Adding first conv layer with 64 filters and kernel size 3x3, padding 'same' provides the output size
# same as the input size. Input_shape denotes input image dimension of MNIST images
model.add(Conv2D(64,(3,3), activation='relu',padding='same', input_shape=(128,128,3)))

# Adding max pooling to reduce the size of output of first conv layer
model.add(MaxPooling2D((2,2),padding='same'))

model.add(Conv2D(32,(3,3), activation='relu', padding='same'))
model.add(MaxPooling2D((2,2),padding='same'))
model.add(Conv2D(32,(3,3),activation='relu',padding='same'))
model.add(MaxPooling2D((2,2),padding='same'))

# Flattening the output of the conv layer after max pooling to make it ready for creating dense connections
model.add(Flatten())

# Adding a fully connected dense layerwith 100 neurons
model.add(Dense(100,activation='relu'))

# Adding the output layer with 10 neurons and activation functions as softmax since this is multi-calssification problem
model.add(Dense(12,activation='softmax'))

#Using SDG Optimizer
opt = SGD(learning_rate=0.01,momentum=0.9)

# Compile Model
model.compile(optimizer=opt, loss='categorical_crossentropy',metrics=['accuracy'])

# Generating the summary of the model
model.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 128, 128, 64)      1792      
                                                                 
 max_pooling2d (MaxPooling2  (None, 64, 64, 64)        0         
 D)                                                              
                                                                 
 conv2d_1 (Conv2D)           (None, 64, 64, 32)        18464     
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 32, 32, 32)        0         
 g2D)                                                            
                                                                 
 conv2d_2 (Conv2D)           (None, 32, 32, 32)        9248      
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 16, 16, 32)        0         
 g2D)                                                            
                                                                 
 flatten (Flatten)           (None, 8192)              0         
                                                                 
 dense (Dense)               (None, 100)               819300    
                                                                 
 dense_1 (Dense)             (None, 12)                1212      
                                                                 
=================================================================
Total params: 850016 (3.24 MB)
Trainable params: 850016 (3.24 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

Let us fit the model on the training data

In [95]:
from keras.callbacks import EarlyStopping, ModelCheckpoint
es = EarlyStopping(monitor='val_loss',mode='min', verbose=1, patience=5)
mc = ModelCheckpoint('best_model.h5',monitor='val_accuracy',mode='max',verbose=1, save_best_only=True)
# Fitting the model with 30 epochs and validation_split as 10%
history=model.fit(X_train,
                  y_train_e,
                  epochs=30,
                  batch_size=32,validation_split=0.10,callbacks=[es,mc])

Epoch 1/30
119/121 [============================>.] - ETA: 0s - loss: 2.4341 - accuracy: 0.1444
Epoch 1: val_accuracy improved from -inf to 0.17523, saving model to best_model.h5
121/121 [==============================] - 4s 24ms/step - loss: 2.4336 - accuracy: 0.1435 - val_loss: 2.3700 - val_accuracy: 0.1752
Epoch 2/30
  7/121 [>.............................] - ETA: 2s - loss: 2.4220 - accuracy: 0.1696
/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

119/121 [============================>.] - ETA: 0s - loss: 2.2287 - accuracy: 0.2379
Epoch 2: val_accuracy improved from 0.17523 to 0.41121, saving model to best_model.h5
121/121 [==============================] - 3s 23ms/step - loss: 2.2250 - accuracy: 0.2389 - val_loss: 1.7588 - val_accuracy: 0.4112
Epoch 3/30
121/121 [==============================] - ETA: 0s - loss: 1.5070 - accuracy: 0.4835
Epoch 3: val_accuracy improved from 0.41121 to 0.60981, saving model to best_model.h5
121/121 [==============================] - 3s 21ms/step - loss: 1.5070 - accuracy: 0.4835 - val_loss: 1.2572 - val_accuracy: 0.6098
Epoch 4/30
121/121 [==============================] - ETA: 0s - loss: 1.1572 - accuracy: 0.6020
Epoch 4: val_accuracy improved from 0.60981 to 0.63785, saving model to best_model.h5
121/121 [==============================] - 3s 23ms/step - loss: 1.1572 - accuracy: 0.6020 - val_loss: 1.1351 - val_accuracy: 0.6379
Epoch 5/30
119/121 [============================>.] - ETA: 0s - loss: 0.9913 - accuracy: 0.6570
Epoch 5: val_accuracy did not improve from 0.63785
121/121 [==============================] - 3s 23ms/step - loss: 0.9913 - accuracy: 0.6571 - val_loss: 1.4791 - val_accuracy: 0.5561
Epoch 6/30
120/121 [============================>.] - ETA: 0s - loss: 0.8363 - accuracy: 0.7122
Epoch 6: val_accuracy improved from 0.63785 to 0.67991, saving model to best_model.h5
121/121 [==============================] - 3s 22ms/step - loss: 0.8362 - accuracy: 0.7122 - val_loss: 1.0192 - val_accuracy: 0.6799
Epoch 7/30
118/121 [============================>.] - ETA: 0s - loss: 0.6619 - accuracy: 0.7712
Epoch 7: val_accuracy improved from 0.67991 to 0.69393, saving model to best_model.h5
121/121 [==============================] - 2s 20ms/step - loss: 0.6644 - accuracy: 0.7697 - val_loss: 1.0967 - val_accuracy: 0.6939
Epoch 8/30
118/121 [============================>.] - ETA: 0s - loss: 0.5723 - accuracy: 0.7977
Epoch 8: val_accuracy improved from 0.69393 to 0.71262, saving model to best_model.h5
121/121 [==============================] - 2s 20ms/step - loss: 0.5732 - accuracy: 0.7988 - val_loss: 0.9135 - val_accuracy: 0.7126
Epoch 9/30
118/121 [============================>.] - ETA: 0s - loss: 0.4348 - accuracy: 0.8477
Epoch 9: val_accuracy improved from 0.71262 to 0.77570, saving model to best_model.h5
121/121 [==============================] - 2s 20ms/step - loss: 0.4401 - accuracy: 0.8456 - val_loss: 0.8740 - val_accuracy: 0.7757
Epoch 10/30
120/121 [============================>.] - ETA: 0s - loss: 0.3573 - accuracy: 0.8690
Epoch 10: val_accuracy did not improve from 0.77570
121/121 [==============================] - 3s 21ms/step - loss: 0.3577 - accuracy: 0.8690 - val_loss: 0.9658 - val_accuracy: 0.7033
Epoch 11/30
120/121 [============================>.] - ETA: 0s - loss: 0.3361 - accuracy: 0.8836
Epoch 11: val_accuracy did not improve from 0.77570
121/121 [==============================] - 3s 23ms/step - loss: 0.3361 - accuracy: 0.8835 - val_loss: 1.0867 - val_accuracy: 0.7336
Epoch 12/30
119/121 [============================>.] - ETA: 0s - loss: 0.2984 - accuracy: 0.8936
Epoch 12: val_accuracy did not improve from 0.77570
121/121 [==============================] - 3s 22ms/step - loss: 0.2974 - accuracy: 0.8939 - val_loss: 0.8773 - val_accuracy: 0.7664
Epoch 13/30
118/121 [============================>.] - ETA: 0s - loss: 0.1374 - accuracy: 0.9523
Epoch 13: val_accuracy did not improve from 0.77570
121/121 [==============================] - 2s 21ms/step - loss: 0.1374 - accuracy: 0.9519 - val_loss: 1.1081 - val_accuracy: 0.7570
Epoch 14/30
120/121 [============================>.] - ETA: 0s - loss: 0.1247 - accuracy: 0.9552
Epoch 14: val_accuracy did not improve from 0.77570
121/121 [==============================] - 2s 21ms/step - loss: 0.1245 - accuracy: 0.9553 - val_loss: 1.2681 - val_accuracy: 0.7734
Epoch 14: early stopping

Plotting Accuracy vs Epoch curve

In [96]:
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model_accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train','test'],loc='upper left')
plt.show()
In [97]:
model.evaluate(X_test,(y_test_e))

15/15 [==============================] - 0s 9ms/step - loss: 1.2502 - accuracy: 0.7453
Out[97]:
[1.2502117156982422, 0.7452631592750549]
In [98]:
# Test Prediction
y_test_pred_ln = model.predict(X_test)
y_test_pred_classes_ln = np.argmax(y_test_pred_ln, axis=1)
normal_y_test = np.argmax(y_test_e, axis=1)

15/15 [==============================] - 0s 9ms/step
In [99]:
# Test Accuracy
import seaborn as sns
from sklearn.metrics import accuracy_score, confusion_matrix
accuracy_score((normal_y_test),y_test_pred_classes_ln)
Out[99]:
0.7452631578947368
In [100]:
# Let us print the sns heatmap

CATEGORIES = list(unique_labels)
cf_matrix = confusion_matrix(normal_y_test, y_test_pred_classes_ln)

# Confusion matrix normalized per category true value
cf_matrix_n1 = cf_matrix/np.sum(cf_matrix, axis=1)
plt.figure(figsize=(8,6))
sns.heatmap(cf_matrix_n1, xticklabels=CATEGORIES, yticklabels=CATEGORIES, annot=True)
Out[100]:
<Axes: >

Observations:

  • Black-grass, Maize, and Common Wheat has most inaccurate predictions.
  • Also these three classes has least data.
  • Adding more images of these classes should certainly help in increasing the prediction accuracy

Model 2: ( More Layers)¶

Let try to build another CNN model with more layers added to the model

In [101]:
from tensorflow.keras import backend
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
In [102]:
# initialized a sequential model
model_3 = Sequential()
# adding first conv layer with 256 filters and kernel size 5x5 , with ReLU activation and padding 'same' provides the output size same as the input size
#input_shape denotes input image dimension of images
model_3.add(Conv2D(filters = 256, kernel_size = (3,3),padding = 'Same',
                 activation ='relu', input_shape = (128,128,3)))
# adding max pooling to reduce the size of output of first conv layer
model_3.add(MaxPool2D(pool_size=(2,2)))
#  adding dropout to randomly switch off 25% neurons to reduce overfitting

# adding second conv layer with 256 filters and with kernel size 3x3 and ReLu activation function
model_3.add(Conv2D(filters = 128, kernel_size = (3,3),padding = 'Same',
                 activation ='relu'))
# adding max pooling to reduce the size of output of first conv layer
model_3.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))
#  adding dropout to randomly switch off 25% neurons to reduce overfitting

# adding third conv layer with 256 filters and with kernel size 3x3 and ReLu activation function
model_3.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same',
                 activation ='relu'))
# adding max pooling to reduce the size of output of first conv layer
model_3.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))
#  adding dropout to randomly switch off 30% neurons to reduce overfitting
model_3.add(Dropout(0.3))

# adding forth conv layer with 256 filters and with kernel size 3x3 and ReLu activation function
model_3.add(Conv2D(filters = 32, kernel_size = (3,3),padding = 'Same',
                 activation ='relu'))
# adding max pooling to reduce the size of output of first conv layer
model_3.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))
#  adding dropout to randomly switch off 30% neurons to reduce overfitting
model_3.add(Dropout(0.3))


# flattening the 3-d output of the conv layer after max pooling to make it ready for creating dense connections
model_3.add(Flatten())
# adding first fully connected dense layer with 1024 neurons
model_3.add(Dense(64, activation = "relu"))
#  adding dropout to randomly switch off 50% neurons to reduce overfitting
# model_3.add(Dropout(0.5))
# adding second fully connected dense layer with 512 neurons
model_3.add(Dense(32, activation = "relu"))
#  adding dropout to randomly switch off 50% neurons to reduce overfitting
#model_3.add(Dropout(0.5))

# adding the output layer with 3 neurons and activation functions as softmax since this is a multi-class classification problem with 3 classes.
model_3.add(Dense(12, activation = "softmax"))
In [103]:
# Let us print the model summary
model_3.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 128, 128, 256)     7168      
                                                                 
 max_pooling2d (MaxPooling2  (None, 64, 64, 256)       0         
 D)                                                              
                                                                 
 conv2d_1 (Conv2D)           (None, 64, 64, 128)       295040    
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 32, 32, 128)       0         
 g2D)                                                            
                                                                 
 conv2d_2 (Conv2D)           (None, 32, 32, 64)        73792     
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 16, 16, 64)        0         
 g2D)                                                            
                                                                 
 dropout (Dropout)           (None, 16, 16, 64)        0         
                                                                 
 conv2d_3 (Conv2D)           (None, 16, 16, 32)        18464     
                                                                 
 max_pooling2d_3 (MaxPoolin  (None, 8, 8, 32)          0         
 g2D)                                                            
                                                                 
 dropout_1 (Dropout)         (None, 8, 8, 32)          0         
                                                                 
 flatten (Flatten)           (None, 2048)              0         
                                                                 
 dense (Dense)               (None, 64)                131136    
                                                                 
 dense_1 (Dense)             (None, 32)                2080      
                                                                 
 dense_2 (Dense)             (None, 12)                396       
                                                                 
=================================================================
Total params: 528076 (2.01 MB)
Trainable params: 528076 (2.01 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [104]:
# Let us defind the early stopping, ModelCheckPoint variables and fit the mode on the training data
optimizer = Adam(lr=0.001)
model_3.compile(optimizer = optimizer , loss = "categorical_crossentropy", metrics=["accuracy"])
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=8)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)

history=model_3.fit(X_train,
          y_train_e,
          epochs=30,
          batch_size=32,validation_split=0.10,callbacks=[es, mc],use_multiprocessing=True)

WARNING:absl:`lr` is deprecated in Keras optimizer, please use `learning_rate` or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.Adam.

Epoch 1/30
120/121 [============================>.] - ETA: 0s - loss: 2.4443 - accuracy: 0.1237
Epoch 1: val_accuracy improved from -inf to 0.16121, saving model to best_model.h5
121/121 [==============================] - 16s 108ms/step - loss: 2.4438 - accuracy: 0.1243 - val_loss: 2.3810 - val_accuracy: 0.1612
Epoch 2/30
  1/121 [..............................] - ETA: 11s - loss: 2.5613 - accuracy: 0.0312
/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

120/121 [============================>.] - ETA: 0s - loss: 2.4271 - accuracy: 0.1318
Epoch 2: val_accuracy did not improve from 0.16121
121/121 [==============================] - 12s 99ms/step - loss: 2.4271 - accuracy: 0.1315 - val_loss: 2.4086 - val_accuracy: 0.1355
Epoch 3/30
120/121 [============================>.] - ETA: 0s - loss: 2.4202 - accuracy: 0.1391
Epoch 3: val_accuracy did not improve from 0.16121
121/121 [==============================] - 12s 100ms/step - loss: 2.4204 - accuracy: 0.1388 - val_loss: 2.3821 - val_accuracy: 0.1472
Epoch 4/30
120/121 [============================>.] - ETA: 0s - loss: 1.9083 - accuracy: 0.3570
Epoch 4: val_accuracy improved from 0.16121 to 0.39953, saving model to best_model.h5
121/121 [==============================] - 12s 99ms/step - loss: 1.9061 - accuracy: 0.3579 - val_loss: 1.6946 - val_accuracy: 0.3995
Epoch 5/30
120/121 [============================>.] - ETA: 0s - loss: 1.4454 - accuracy: 0.4862
Epoch 5: val_accuracy improved from 0.39953 to 0.60748, saving model to best_model.h5
121/121 [==============================] - 12s 95ms/step - loss: 1.4454 - accuracy: 0.4866 - val_loss: 1.1850 - val_accuracy: 0.6075
Epoch 6/30
120/121 [============================>.] - ETA: 0s - loss: 1.1217 - accuracy: 0.6112
Epoch 6: val_accuracy improved from 0.60748 to 0.66121, saving model to best_model.h5
121/121 [==============================] - 12s 95ms/step - loss: 1.1222 - accuracy: 0.6111 - val_loss: 1.1157 - val_accuracy: 0.6612
Epoch 7/30
120/121 [============================>.] - ETA: 0s - loss: 0.9611 - accuracy: 0.6677
Epoch 7: val_accuracy improved from 0.66121 to 0.74065, saving model to best_model.h5
121/121 [==============================] - 12s 96ms/step - loss: 0.9609 - accuracy: 0.6675 - val_loss: 0.9568 - val_accuracy: 0.7407
Epoch 8/30
120/121 [============================>.] - ETA: 0s - loss: 0.8706 - accuracy: 0.7023
Epoch 8: val_accuracy did not improve from 0.74065
121/121 [==============================] - 11s 95ms/step - loss: 0.8702 - accuracy: 0.7018 - val_loss: 0.8355 - val_accuracy: 0.7360
Epoch 9/30
120/121 [============================>.] - ETA: 0s - loss: 0.7484 - accuracy: 0.7378
Epoch 9: val_accuracy did not improve from 0.74065
121/121 [==============================] - 11s 95ms/step - loss: 0.7481 - accuracy: 0.7377 - val_loss: 0.9552 - val_accuracy: 0.7126
Epoch 10/30
120/121 [============================>.] - ETA: 0s - loss: 0.6971 - accuracy: 0.7578
Epoch 10: val_accuracy improved from 0.74065 to 0.79206, saving model to best_model.h5
121/121 [==============================] - 12s 96ms/step - loss: 0.6968 - accuracy: 0.7580 - val_loss: 0.7605 - val_accuracy: 0.7921
Epoch 11/30
120/121 [============================>.] - ETA: 0s - loss: 0.6311 - accuracy: 0.7789
Epoch 11: val_accuracy did not improve from 0.79206
121/121 [==============================] - 12s 96ms/step - loss: 0.6304 - accuracy: 0.7793 - val_loss: 0.7351 - val_accuracy: 0.7780
Epoch 12/30
120/121 [============================>.] - ETA: 0s - loss: 0.5713 - accuracy: 0.8055
Epoch 12: val_accuracy improved from 0.79206 to 0.82710, saving model to best_model.h5
121/121 [==============================] - 11s 95ms/step - loss: 0.5716 - accuracy: 0.8053 - val_loss: 0.6542 - val_accuracy: 0.8271
Epoch 13/30
120/121 [============================>.] - ETA: 0s - loss: 0.5302 - accuracy: 0.8096
Epoch 13: val_accuracy did not improve from 0.82710
121/121 [==============================] - 12s 96ms/step - loss: 0.5312 - accuracy: 0.8092 - val_loss: 0.6726 - val_accuracy: 0.7991
Epoch 14/30
120/121 [============================>.] - ETA: 0s - loss: 0.4717 - accuracy: 0.8271
Epoch 14: val_accuracy did not improve from 0.82710
121/121 [==============================] - 12s 95ms/step - loss: 0.4716 - accuracy: 0.8271 - val_loss: 0.6769 - val_accuracy: 0.8014
Epoch 15/30
120/121 [============================>.] - ETA: 0s - loss: 0.4376 - accuracy: 0.8469
Epoch 15: val_accuracy did not improve from 0.82710
121/121 [==============================] - 11s 95ms/step - loss: 0.4371 - accuracy: 0.8472 - val_loss: 0.7460 - val_accuracy: 0.7874
Epoch 16/30
120/121 [============================>.] - ETA: 0s - loss: 0.3894 - accuracy: 0.8646
Epoch 16: val_accuracy did not improve from 0.82710
121/121 [==============================] - 12s 95ms/step - loss: 0.3890 - accuracy: 0.8648 - val_loss: 0.7398 - val_accuracy: 0.7827
Epoch 17/30
120/121 [============================>.] - ETA: 0s - loss: 0.3818 - accuracy: 0.8648
Epoch 17: val_accuracy did not improve from 0.82710
121/121 [==============================] - 12s 96ms/step - loss: 0.3821 - accuracy: 0.8646 - val_loss: 0.7657 - val_accuracy: 0.8014
Epoch 18/30
120/121 [============================>.] - ETA: 0s - loss: 0.3512 - accuracy: 0.8792
Epoch 18: val_accuracy did not improve from 0.82710
121/121 [==============================] - 11s 94ms/step - loss: 0.3513 - accuracy: 0.8791 - val_loss: 0.7402 - val_accuracy: 0.8084
Epoch 19/30
120/121 [============================>.] - ETA: 0s - loss: 0.3337 - accuracy: 0.8784
Epoch 19: val_accuracy did not improve from 0.82710
121/121 [==============================] - 11s 94ms/step - loss: 0.3344 - accuracy: 0.8781 - val_loss: 0.7441 - val_accuracy: 0.8131
Epoch 20/30
120/121 [============================>.] - ETA: 0s - loss: 0.3093 - accuracy: 0.8875
Epoch 20: val_accuracy did not improve from 0.82710
121/121 [==============================] - 12s 96ms/step - loss: 0.3089 - accuracy: 0.8877 - val_loss: 0.7785 - val_accuracy: 0.7804
Epoch 20: early stopping

Plotting Accuracy vs Epoch Curve

In [105]:
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'Val'], loc='upper left')
plt.show()

Train and Validation accuracy seems fine and let's calculate the accuracy on the test data

In [106]:
model_3.evaluate(X_test,y_test_e)

15/15 [==============================] - 0s 24ms/step - loss: 0.8055 - accuracy: 0.7768
Out[106]:
[0.8055254220962524, 0.7768421173095703]
In [107]:
y_test_pred_ln3 = model_3.predict(X_test)
y_test_pred_classes_ln3 = np.argmax(y_test_pred_ln3, axis=1)

15/15 [==============================] - 0s 17ms/step
In [108]:
import seaborn as sns
from sklearn.metrics import accuracy_score, confusion_matrix
accuracy_score(normal_y_test, y_test_pred_classes_ln3)
Out[108]:
0.7768421052631579
In [109]:
cf_matrix = confusion_matrix(normal_y_test, y_test_pred_classes_ln3)

# Confusion matrix normalized per category true value
cf_matrix_n1 = cf_matrix/np.sum(cf_matrix, axis=1)
plt.figure(figsize=(8,6))
sns.heatmap(cf_matrix_n1, xticklabels=CATEGORIES, yticklabels=CATEGORIES, annot=True)
Out[109]:
<Axes: >

Observations:

  • We got around 83% of correct prediction which is good
  • All the classes except Black-grass has good prediction.
  • Maize and Common Wheat has good prediction eventhough the data is less
  • Adding more images of Black-grass in the test pool should further improve the performance of the model

Model: 3 With Data Augmentation¶

Let us try with Data Agumentation

Displaying images before augmentation

In [110]:
num_images = 5
plt.figure(figsize=(15, 15))
for i in range(num_images):
    plt.subplot(1, num_images, i + 1)
    plt.imshow(X_train[i])

    plt.axis('off')

plt.show()

Augment the data using ImageDataGenerator class¶

In [111]:
# All images to be rescaled by 1/255.
train_datagen = ImageDataGenerator(
    rescale=1./255,  # normalize pixel values to [0,1]
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
)
test_datagen  = ImageDataGenerator(rescale = 1.0/255.)

X_train = X_train.astype('float32')
X_test = X_test.astype('float32')

# Fit the generators on the training data
train_datagen.fit(X_train)

# Generate augmented images for training
train_generator = train_datagen.flow(X_train, y_train_e, batch_size=24,
                                                    shuffle=True)

# Generate rescaled images for testing (without augmentation)
test_generator = test_datagen.flow(X_test, y_test_e, batch_size=24,
                                                    shuffle=False)

Let us display the images after augmentation¶

In [112]:
# Number of images to display
num_images = 5

# Obtain a batch of augmented images and labels from the generator
augmented_images, augmented_labels = train_generator.next()

# Display the images
plt.figure(figsize=(15, 15))
for i in range(num_images):
    plt.subplot(1, num_images, i + 1)
    plt.imshow(augmented_images[i]*255)

    plt.axis('off')

plt.show()

Let us build the model

In [113]:
cnn_model = Sequential()
cnn_model.add(Conv2D(128, (3,3), activation='relu', input_shape=(128, 128, 3), padding = 'same'))
cnn_model.add(MaxPooling2D(2,2))
cnn_model.add(BatchNormalization())
cnn_model.add(Conv2D(64, (3,3), activation='relu', padding = 'same'))
cnn_model.add(MaxPooling2D(2,2))
cnn_model.add(BatchNormalization())
cnn_model.add(Conv2D(64, (3,3), activation='relu', padding = 'same'))
cnn_model.add(MaxPooling2D(2,2))
cnn_model.add(Conv2D(16, (3,3), activation='relu', padding = 'same'))
cnn_model.add(Flatten())
cnn_model.add(Dense(64, activation='relu'))
cnn_model.add(Dropout(0.25))
cnn_model.add(Dense(32, activation='relu'))
cnn_model.add(Dropout(0.25))
cnn_model.add(Dense(32, activation='relu'))
cnn_model.add(Dense(12, activation='softmax'))
In [114]:
# Compiling the model
cnn_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
# Summary of the model
cnn_model.summary()

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d_4 (Conv2D)           (None, 128, 128, 128)     3584      
                                                                 
 max_pooling2d_4 (MaxPoolin  (None, 64, 64, 128)       0         
 g2D)                                                            
                                                                 
 batch_normalization (Batch  (None, 64, 64, 128)       512       
 Normalization)                                                  
                                                                 
 conv2d_5 (Conv2D)           (None, 64, 64, 64)        73792     
                                                                 
 max_pooling2d_5 (MaxPoolin  (None, 32, 32, 64)        0         
 g2D)                                                            
                                                                 
 batch_normalization_1 (Bat  (None, 32, 32, 64)        256       
 chNormalization)                                                
                                                                 
 conv2d_6 (Conv2D)           (None, 32, 32, 64)        36928     
                                                                 
 max_pooling2d_6 (MaxPoolin  (None, 16, 16, 64)        0         
 g2D)                                                            
                                                                 
 conv2d_7 (Conv2D)           (None, 16, 16, 16)        9232      
                                                                 
 flatten_1 (Flatten)         (None, 4096)              0         
                                                                 
 dense_3 (Dense)             (None, 64)                262208    
                                                                 
 dropout_2 (Dropout)         (None, 64)                0         
                                                                 
 dense_4 (Dense)             (None, 32)                2080      
                                                                 
 dropout_3 (Dropout)         (None, 32)                0         
                                                                 
 dense_5 (Dense)             (None, 32)                1056      
                                                                 
 dense_6 (Dense)             (None, 12)                396       
                                                                 
=================================================================
Total params: 390044 (1.49 MB)
Trainable params: 389660 (1.49 MB)
Non-trainable params: 384 (1.50 KB)
_________________________________________________________________
In [115]:
## Pulling a single large batch of random validation data for testing after each epoch
testX, testY = test_generator.next()

Let us fit the model with augmented data and check the performance

In [116]:
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=5)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)

## Fitting the Aug model
cnn_model_history = cnn_model.fit(train_generator,
                                  validation_data = (testX, testY),
                                  epochs=30,callbacks=[es, mc],use_multiprocessing=True)

Epoch 1/30
179/179 [==============================] - ETA: 0s - loss: 2.3233 - accuracy: 0.1977
Epoch 1: val_accuracy improved from -inf to 0.12500, saving model to best_model.h5
179/179 [==============================] - 44s 215ms/step - loss: 2.3233 - accuracy: 0.1977 - val_loss: 2.6576 - val_accuracy: 0.1250

/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

Epoch 2/30
179/179 [==============================] - ETA: 0s - loss: 1.9627 - accuracy: 0.2924
Epoch 2: val_accuracy did not improve from 0.12500
179/179 [==============================] - 44s 242ms/step - loss: 1.9627 - accuracy: 0.2924 - val_loss: 4.6108 - val_accuracy: 0.1250
Epoch 3/30
179/179 [==============================] - ETA: 0s - loss: 1.8166 - accuracy: 0.3570
Epoch 3: val_accuracy did not improve from 0.12500
179/179 [==============================] - 43s 241ms/step - loss: 1.8166 - accuracy: 0.3570 - val_loss: 2.3390 - val_accuracy: 0.1250
Epoch 4/30
179/179 [==============================] - ETA: 0s - loss: 1.6662 - accuracy: 0.4091
Epoch 4: val_accuracy improved from 0.12500 to 0.16667, saving model to best_model.h5
179/179 [==============================] - 45s 251ms/step - loss: 1.6662 - accuracy: 0.4091 - val_loss: 2.1631 - val_accuracy: 0.1667
Epoch 5/30
179/179 [==============================] - ETA: 0s - loss: 1.5827 - accuracy: 0.4374
Epoch 5: val_accuracy did not improve from 0.16667
179/179 [==============================] - 44s 245ms/step - loss: 1.5827 - accuracy: 0.4374 - val_loss: 2.2272 - val_accuracy: 0.1250
Epoch 6/30
179/179 [==============================] - ETA: 0s - loss: 1.5229 - accuracy: 0.4643
Epoch 6: val_accuracy did not improve from 0.16667
179/179 [==============================] - 42s 235ms/step - loss: 1.5229 - accuracy: 0.4643 - val_loss: 7.0037 - val_accuracy: 0.0000e+00
Epoch 7/30
179/179 [==============================] - ETA: 0s - loss: 1.4590 - accuracy: 0.4936
Epoch 7: val_accuracy improved from 0.16667 to 0.54167, saving model to best_model.h5
179/179 [==============================] - 46s 253ms/step - loss: 1.4590 - accuracy: 0.4936 - val_loss: 1.4965 - val_accuracy: 0.5417
Epoch 8/30
179/179 [==============================] - ETA: 0s - loss: 1.3922 - accuracy: 0.4971
Epoch 8: val_accuracy did not improve from 0.54167
179/179 [==============================] - 45s 249ms/step - loss: 1.3922 - accuracy: 0.4971 - val_loss: 1.4369 - val_accuracy: 0.5417
Epoch 9/30
179/179 [==============================] - ETA: 0s - loss: 1.3745 - accuracy: 0.5268
Epoch 9: val_accuracy did not improve from 0.54167
179/179 [==============================] - 45s 249ms/step - loss: 1.3745 - accuracy: 0.5268 - val_loss: 3.6982 - val_accuracy: 0.1667
Epoch 10/30
179/179 [==============================] - ETA: 0s - loss: 1.3085 - accuracy: 0.5427
Epoch 10: val_accuracy improved from 0.54167 to 0.62500, saving model to best_model.h5
179/179 [==============================] - 47s 262ms/step - loss: 1.3085 - accuracy: 0.5427 - val_loss: 1.1831 - val_accuracy: 0.6250
Epoch 11/30
179/179 [==============================] - ETA: 0s - loss: 1.2812 - accuracy: 0.5593
Epoch 11: val_accuracy did not improve from 0.62500
179/179 [==============================] - 43s 238ms/step - loss: 1.2812 - accuracy: 0.5593 - val_loss: 2.1719 - val_accuracy: 0.2500
Epoch 12/30
179/179 [==============================] - ETA: 0s - loss: 1.2428 - accuracy: 0.5680
Epoch 12: val_accuracy did not improve from 0.62500
179/179 [==============================] - 43s 239ms/step - loss: 1.2428 - accuracy: 0.5680 - val_loss: 1.3689 - val_accuracy: 0.5833
Epoch 13/30
179/179 [==============================] - ETA: 0s - loss: 1.2131 - accuracy: 0.5726
Epoch 13: val_accuracy did not improve from 0.62500
179/179 [==============================] - 47s 261ms/step - loss: 1.2131 - accuracy: 0.5726 - val_loss: 1.3967 - val_accuracy: 0.6250
Epoch 14/30
179/179 [==============================] - ETA: 0s - loss: 1.1851 - accuracy: 0.5937
Epoch 14: val_accuracy did not improve from 0.62500
179/179 [==============================] - 43s 240ms/step - loss: 1.1851 - accuracy: 0.5937 - val_loss: 3.3549 - val_accuracy: 0.1250
Epoch 15/30
179/179 [==============================] - ETA: 0s - loss: 1.2075 - accuracy: 0.5780
Epoch 15: val_accuracy did not improve from 0.62500
179/179 [==============================] - 44s 244ms/step - loss: 1.2075 - accuracy: 0.5780 - val_loss: 1.3158 - val_accuracy: 0.5417
Epoch 15: early stopping

Observation:

  • There is no improvment in the accuracy after 62 percentage
  • Data augmentation does not help in increasing the accuracy
  • Will skip this model as the performance is not good

Model 4: Using Transfer Learning¶

In [117]:
# Loading VGG16 model
model = VGG16(weights='imagenet')
# Summary of the whole model
model.summary()

Model: "vgg16"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 224, 224, 3)]     0         
                                                                 
 block1_conv1 (Conv2D)       (None, 224, 224, 64)      1792      
                                                                 
 block1_conv2 (Conv2D)       (None, 224, 224, 64)      36928     
                                                                 
 block1_pool (MaxPooling2D)  (None, 112, 112, 64)      0         
                                                                 
 block2_conv1 (Conv2D)       (None, 112, 112, 128)     73856     
                                                                 
 block2_conv2 (Conv2D)       (None, 112, 112, 128)     147584    
                                                                 
 block2_pool (MaxPooling2D)  (None, 56, 56, 128)       0         
                                                                 
 block3_conv1 (Conv2D)       (None, 56, 56, 256)       295168    
                                                                 
 block3_conv2 (Conv2D)       (None, 56, 56, 256)       590080    
                                                                 
 block3_conv3 (Conv2D)       (None, 56, 56, 256)       590080    
                                                                 
 block3_pool (MaxPooling2D)  (None, 28, 28, 256)       0         
                                                                 
 block4_conv1 (Conv2D)       (None, 28, 28, 512)       1180160   
                                                                 
 block4_conv2 (Conv2D)       (None, 28, 28, 512)       2359808   
                                                                 
 block4_conv3 (Conv2D)       (None, 28, 28, 512)       2359808   
                                                                 
 block4_pool (MaxPooling2D)  (None, 14, 14, 512)       0         
                                                                 
 block5_conv1 (Conv2D)       (None, 14, 14, 512)       2359808   
                                                                 
 block5_conv2 (Conv2D)       (None, 14, 14, 512)       2359808   
                                                                 
 block5_conv3 (Conv2D)       (None, 14, 14, 512)       2359808   
                                                                 
 block5_pool (MaxPooling2D)  (None, 7, 7, 512)         0         
                                                                 
 flatten (Flatten)           (None, 25088)             0         
                                                                 
 fc1 (Dense)                 (None, 4096)              102764544 
                                                                 
 fc2 (Dense)                 (None, 4096)              16781312  
                                                                 
 predictions (Dense)         (None, 1000)              4097000   
                                                                 
=================================================================
Total params: 138357544 (527.79 MB)
Trainable params: 138357544 (527.79 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [118]:
# Getting only the conv layers for transfer learning.
transfer_layer = model.get_layer('block5_pool')
vgg_model = Model(inputs=model.input, outputs=transfer_layer.output)
In [119]:
# Printing the model summary
vgg_model.summary()

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 224, 224, 3)]     0         
                                                                 
 block1_conv1 (Conv2D)       (None, 224, 224, 64)      1792      
                                                                 
 block1_conv2 (Conv2D)       (None, 224, 224, 64)      36928     
                                                                 
 block1_pool (MaxPooling2D)  (None, 112, 112, 64)      0         
                                                                 
 block2_conv1 (Conv2D)       (None, 112, 112, 128)     73856     
                                                                 
 block2_conv2 (Conv2D)       (None, 112, 112, 128)     147584    
                                                                 
 block2_pool (MaxPooling2D)  (None, 56, 56, 128)       0         
                                                                 
 block3_conv1 (Conv2D)       (None, 56, 56, 256)       295168    
                                                                 
 block3_conv2 (Conv2D)       (None, 56, 56, 256)       590080    
                                                                 
 block3_conv3 (Conv2D)       (None, 56, 56, 256)       590080    
                                                                 
 block3_pool (MaxPooling2D)  (None, 28, 28, 256)       0         
                                                                 
 block4_conv1 (Conv2D)       (None, 28, 28, 512)       1180160   
                                                                 
 block4_conv2 (Conv2D)       (None, 28, 28, 512)       2359808   
                                                                 
 block4_conv3 (Conv2D)       (None, 28, 28, 512)       2359808   
                                                                 
 block4_pool (MaxPooling2D)  (None, 14, 14, 512)       0         
                                                                 
 block5_conv1 (Conv2D)       (None, 14, 14, 512)       2359808   
                                                                 
 block5_conv2 (Conv2D)       (None, 14, 14, 512)       2359808   
                                                                 
 block5_conv3 (Conv2D)       (None, 14, 14, 512)       2359808   
                                                                 
 block5_pool (MaxPooling2D)  (None, 7, 7, 512)         0         
                                                                 
=================================================================
Total params: 14714688 (56.13 MB)
Trainable params: 14714688 (56.13 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [120]:
vgg_model = VGG16(weights='imagenet', include_top = False, input_shape = (128,128,3))
vgg_model.summary()

Model: "vgg16"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_2 (InputLayer)        [(None, 128, 128, 3)]     0         
                                                                 
 block1_conv1 (Conv2D)       (None, 128, 128, 64)      1792      
                                                                 
 block1_conv2 (Conv2D)       (None, 128, 128, 64)      36928     
                                                                 
 block1_pool (MaxPooling2D)  (None, 64, 64, 64)        0         
                                                                 
 block2_conv1 (Conv2D)       (None, 64, 64, 128)       73856     
                                                                 
 block2_conv2 (Conv2D)       (None, 64, 64, 128)       147584    
                                                                 
 block2_pool (MaxPooling2D)  (None, 32, 32, 128)       0         
                                                                 
 block3_conv1 (Conv2D)       (None, 32, 32, 256)       295168    
                                                                 
 block3_conv2 (Conv2D)       (None, 32, 32, 256)       590080    
                                                                 
 block3_conv3 (Conv2D)       (None, 32, 32, 256)       590080    
                                                                 
 block3_pool (MaxPooling2D)  (None, 16, 16, 256)       0         
                                                                 
 block4_conv1 (Conv2D)       (None, 16, 16, 512)       1180160   
                                                                 
 block4_conv2 (Conv2D)       (None, 16, 16, 512)       2359808   
                                                                 
 block4_conv3 (Conv2D)       (None, 16, 16, 512)       2359808   
                                                                 
 block4_pool (MaxPooling2D)  (None, 8, 8, 512)         0         
                                                                 
 block5_conv1 (Conv2D)       (None, 8, 8, 512)         2359808   
                                                                 
 block5_conv2 (Conv2D)       (None, 8, 8, 512)         2359808   
                                                                 
 block5_conv3 (Conv2D)       (None, 8, 8, 512)         2359808   
                                                                 
 block5_pool (MaxPooling2D)  (None, 4, 4, 512)         0         
                                                                 
=================================================================
Total params: 14714688 (56.13 MB)
Trainable params: 14714688 (56.13 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [121]:
# Making all the layers of the VGG model non-trainable. i.e. freezing them
for layer in vgg_model.layers:
    layer.trainable = False
In [122]:
for layer in vgg_model.layers:
    print(layer.name, layer.trainable)

input_2 False
block1_conv1 False
block1_conv2 False
block1_pool False
block2_conv1 False
block2_conv2 False
block2_pool False
block3_conv1 False
block3_conv2 False
block3_conv3 False
block3_pool False
block4_conv1 False
block4_conv2 False
block4_conv3 False
block4_pool False
block5_conv1 False
block5_conv2 False
block5_conv3 False
block5_pool False

Let us start building the model

In [123]:
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
In [124]:
# Initializing the model
new_model = Sequential()

# Adding the convolutional part of the VGG16 model from above
new_model.add(vgg_model)

# Flattening the output of the VGG16 model because it is from a convolutional layer
new_model.add(Flatten())

# Adding a dense input layer
new_model.add(Dense(128, activation='relu'))
# Adding dropout
new_model.add(Dropout(0.2))
# Adding second input layer
new_model.add(Dense(128, activation='relu'))
# Adding output layer
new_model.add(Dense(12, activation='softmax'))
In [125]:
# Compiling the model
new_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
# Summary of the model
new_model.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 vgg16 (Functional)          (None, 4, 4, 512)         14714688  
                                                                 
 flatten (Flatten)           (None, 8192)              0         
                                                                 
 dense (Dense)               (None, 128)               1048704   
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_1 (Dense)             (None, 128)               16512     
                                                                 
 dense_2 (Dense)             (None, 12)                1548      
                                                                 
=================================================================
Total params: 15781452 (60.20 MB)
Trainable params: 1066764 (4.07 MB)
Non-trainable params: 14714688 (56.13 MB)
_________________________________________________________________
In [126]:
## Pulling a single large batch of random validation data for testing after each epoch
testX, testY = test_generator.next()
In [127]:
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=5)
mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)

## Fitting the VGG model
#new_model_history = new_model.fit(train_generator,
 #                                 validation_data = (testX, testY),
  #                                epochs=30,callbacks=[es, mc],use_multiprocessing=True)


new_model_history=new_model.fit(X_train,
          y_train_e,
          epochs=30,
          batch_size=32,validation_split=0.10,callbacks=[es, mc],use_multiprocessing=True)

Epoch 1/30
120/121 [============================>.] - ETA: 0s - loss: 1.9156 - accuracy: 0.3484
Epoch 1: val_accuracy improved from -inf to 0.44626, saving model to best_model.h5
121/121 [==============================] - 9s 55ms/step - loss: 1.9150 - accuracy: 0.3486 - val_loss: 1.5390 - val_accuracy: 0.4463
Epoch 2/30
121/121 [==============================] - ETA: 0s - loss: 1.3724 - accuracy: 0.5157
Epoch 2: val_accuracy improved from 0.44626 to 0.55374, saving model to best_model.h5
121/121 [==============================] - 7s 55ms/step - loss: 1.3724 - accuracy: 0.5157 - val_loss: 1.2697 - val_accuracy: 0.5537
Epoch 3/30
120/121 [============================>.] - ETA: 0s - loss: 1.0423 - accuracy: 0.6393
Epoch 3: val_accuracy improved from 0.55374 to 0.69860, saving model to best_model.h5
121/121 [==============================] - 7s 54ms/step - loss: 1.0414 - accuracy: 0.6397 - val_loss: 0.9807 - val_accuracy: 0.6986
Epoch 4/30
121/121 [==============================] - ETA: 0s - loss: 0.9000 - accuracy: 0.6914
Epoch 4: val_accuracy improved from 0.69860 to 0.70327, saving model to best_model.h5
121/121 [==============================] - 7s 54ms/step - loss: 0.9000 - accuracy: 0.6914 - val_loss: 0.9205 - val_accuracy: 0.7033
Epoch 5/30
120/121 [============================>.] - ETA: 0s - loss: 0.7580 - accuracy: 0.7354
Epoch 5: val_accuracy improved from 0.70327 to 0.72196, saving model to best_model.h5
121/121 [==============================] - 6s 54ms/step - loss: 0.7578 - accuracy: 0.7356 - val_loss: 0.8349 - val_accuracy: 0.7220
Epoch 6/30
120/121 [============================>.] - ETA: 0s - loss: 0.6972 - accuracy: 0.7542
Epoch 6: val_accuracy improved from 0.72196 to 0.76168, saving model to best_model.h5
121/121 [==============================] - 6s 52ms/step - loss: 0.6968 - accuracy: 0.7544 - val_loss: 0.8056 - val_accuracy: 0.7617
Epoch 7/30
120/121 [============================>.] - ETA: 0s - loss: 0.5997 - accuracy: 0.7888
Epoch 7: val_accuracy did not improve from 0.76168
121/121 [==============================] - 6s 51ms/step - loss: 0.5995 - accuracy: 0.7887 - val_loss: 0.8200 - val_accuracy: 0.7383
Epoch 8/30
120/121 [============================>.] - ETA: 0s - loss: 0.5646 - accuracy: 0.8036
Epoch 8: val_accuracy did not improve from 0.76168
121/121 [==============================] - 6s 50ms/step - loss: 0.5639 - accuracy: 0.8040 - val_loss: 0.8450 - val_accuracy: 0.7290
Epoch 9/30
121/121 [==============================] - ETA: 0s - loss: 0.5178 - accuracy: 0.8162
Epoch 9: val_accuracy did not improve from 0.76168
121/121 [==============================] - 6s 49ms/step - loss: 0.5178 - accuracy: 0.8162 - val_loss: 0.8209 - val_accuracy: 0.7313
Epoch 10/30
120/121 [============================>.] - ETA: 0s - loss: 0.4687 - accuracy: 0.8378
Epoch 10: val_accuracy did not improve from 0.76168
121/121 [==============================] - 6s 51ms/step - loss: 0.4687 - accuracy: 0.8378 - val_loss: 0.8398 - val_accuracy: 0.7336
Epoch 11/30
120/121 [============================>.] - ETA: 0s - loss: 0.4576 - accuracy: 0.8393
Epoch 11: val_accuracy improved from 0.76168 to 0.76402, saving model to best_model.h5
121/121 [==============================] - 6s 51ms/step - loss: 0.4579 - accuracy: 0.8391 - val_loss: 0.8141 - val_accuracy: 0.7640
Epoch 11: early stopping

Plotting Accuracy vs Epoch Curve

In [128]:
plt.plot(new_model_history.history['accuracy'])
plt.plot(new_model_history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'Val'], loc='upper left')
plt.show()
In [129]:
# Evaluating on the Test set
new_model.evaluate(test_generator)

20/20 [==============================] - 1s 40ms/step - loss: 5.0914 - accuracy: 0.1368
Out[129]:
[5.091447830200195, 0.13684210181236267]
In [130]:
# Test Prediction
y_test_pred_ln4 = new_model.predict(X_test)
y_test_pred_classes_ln4 = np.argmax(y_test_pred_ln4, axis=1)

15/15 [==============================] - 1s 41ms/step
In [131]:
# Test Accuracy
import seaborn as sns
from sklearn.metrics import accuracy_score, confusion_matrix
accuracy_score(normal_y_test, y_test_pred_classes_ln4)
Out[131]:
0.7326315789473684
In [132]:
# Let us print the sns heatmap
cf_matrix = confusion_matrix(normal_y_test, y_test_pred_classes_ln4)

# Confusion matrix normalized per category true value
cf_matrix_n1 = cf_matrix/np.sum(cf_matrix, axis=1)
plt.figure(figsize=(8,6))
sns.heatmap(cf_matrix_n1, xticklabels=CATEGORIES, yticklabels=CATEGORIES, annot=True)
Out[132]:
<Axes: >

Observation:

  • Accuracy is 73% with the transfer model
  • This model, is able to identify all the classes fairly well compared to the other models
  • On average, the performance for all classes is well improved

Classification Report for each class

  • Precision: precision is the fraction of relevant instances among the retrieved instances.

  • Recall: recall is the fraction of relevant instances that were retrieved.

  • F1 score: The F1 score is the harmonic mean of precision and recall, reaching its optimal value at 1 and its worst value at 0.

Model 1 ( Limited Layers )

In [133]:
from sklearn.metrics import classification_report
print(classification_report((normal_y_test), y_test_pred_classes_ln))

              precision    recall  f1-score   support

           0       1.00      0.04      0.07        26
           1       0.91      0.74      0.82        39
           2       0.88      0.76      0.81        29
           3       0.76      0.90      0.83        61
           4       0.64      0.64      0.64        22
           5       0.85      0.58      0.69        48
           6       0.62      0.91      0.74        65
           7       0.83      0.68      0.75        22
           8       0.64      0.90      0.75        52
           9       0.65      0.65      0.65        23
          10       0.93      0.76      0.84        50
          11       0.78      0.82      0.79        38

    accuracy                           0.75       475
   macro avg       0.79      0.70      0.70       475
weighted avg       0.78      0.75      0.73       475

Model 2 ( Additional Layers)

In [134]:
from sklearn.metrics import classification_report
print(classification_report((normal_y_test), y_test_pred_classes_ln3))

              precision    recall  f1-score   support

           0       0.40      0.38      0.39        26
           1       0.83      0.90      0.86        39
           2       0.68      0.93      0.78        29
           3       0.95      0.85      0.90        61
           4       0.71      0.77      0.74        22
           5       0.82      0.67      0.74        48
           6       0.78      0.66      0.72        65
           7       0.78      0.82      0.80        22
           8       0.74      0.88      0.81        52
           9       0.78      0.61      0.68        23
          10       0.86      0.88      0.87        50
          11       0.76      0.82      0.78        38

    accuracy                           0.78       475
   macro avg       0.76      0.76      0.76       475
weighted avg       0.78      0.78      0.77       475

Model 4 (With Transfer)

In [135]:
from sklearn.metrics import classification_report
print(classification_report((normal_y_test), y_test_pred_classes_ln4))

              precision    recall  f1-score   support

           0       0.48      0.46      0.47        26
           1       0.86      0.79      0.83        39
           2       0.73      0.83      0.77        29
           3       0.91      0.84      0.87        61
           4       0.58      0.64      0.61        22
           5       0.76      0.54      0.63        48
           6       0.65      0.86      0.74        65
           7       0.84      0.73      0.78        22
           8       0.72      0.79      0.75        52
           9       0.67      0.43      0.53        23
          10       0.78      0.80      0.79        50
          11       0.69      0.71      0.70        38

    accuracy                           0.73       475
   macro avg       0.72      0.70      0.71       475
weighted avg       0.74      0.73      0.73       475

Final Model¶

  • Model 2 ( With Additional Layers) is the final model as it provides better accuracy than other two models

Actionable Insights and Business Recommendations¶

  • Efficiency improvement: AI and CNN can signficantly reduce the time required for plant seedling identification compared to manual labor. Invest in AI solutions to streamline the identification process, improving overall efficiency in plant management and monitoring

  • Labor Optimization: By automating the identitifacation of plan seedlings, manual labor can be redirected towards higher order decision making tasks. Implment AI technologies to augment human effort allowing skilled labor to focus on strategic planning, problem-solving and decision-making for good farm management.

  • Enhanced Crop management: AI driven identification can lead better crop management practices resuling in improved yiedls. Integrate AI technolies into crop management system to optimize planting, watering, and harvesting practices to enhance the overall agricultural productivity

*